1. Field of the Invention
The invention relates to an image processing apparatus and its method for performing a separating and synthesizing process of a background and a foreground for a motion image.
2. Related Background Art
In recent years, in association with the spread of a personal computer in which an advanced CPU is installed, a request for an edition of a motion image which is executed on the personal computer has been increasing. As examples of an editing work, there are various works such as exchange of the time order between frames or fields, wiping, dissolving, mosaic, insertion of another image, and the like. Instead of processing an image on a frame or field unit basis, a technique for separating an image into meaning units (hereinafter, called objects) such as objects, backgrounds, characters, or the like in the image and performing an individual process is also being improved. By changing an encoding system or encoding parameter every object, a high efficient transmission or recording in which error withstanding performance is enhanced can be also performed. To perform the individual process on an object unit basis, the object has to be extracted from a frame or field image.
An object extracting method which has conventionally been used with respect to a motion image is called a xe2x80x9cblue backxe2x80x9d. According to the blue back, a blue background is prepared in a studio set or the like and the blue portion is replaced with another background image by a switcher. As a method which is frequently used in a still image, a method of detecting and extracting an edge portion, a method of extracting by providing a threshold value for a signal level, or the like has been known. However, the conventional method using the blue back has a problem such that if a blue picture pattern exists in an area other than the background portion, such an area is erroneously recognized as a background. There is also a problem such that it is necessary to prepare studio equipment. Even in a digital process which can solve the above drawbacks, since it takes a long time for arithmetic operations, there is a problem such that in case of adapting to a motion image, real-time performance has to be sacrificed.
On the other hand, in association with the recent realization of high fineness of an image, the contents of information which the image has are changing. For example, there is a case where characters are superimposed onto a motion image and a resultant image is transmitted or a case where another image is superimposed to the motion image and the resultant image is transmitted. An amount of information which can be transmitted per unit time is increasing. The necessity to extract only necessary portions from a plurality of information and to store or re-edit them will further increase in future.
In case of separating a background object and a foreground object, however, there is hardly a case where a boundary portion of the object is clearly separated on a pixel unit basis. A blur area which is caused due to optical characteristics of an image pickup device exists and pixels in such a blur area are in a state where signal levels of the background and foreground are mixed. Such a situation is particularly typical with respect to a motion object. It is, therefore, important how to handle such a vague boundary area for a process of the object unit.
The problems to be solved by the invention will now be described in detail hereinbelow with reference to the drawings.
FIG. 1A shows an example of an original image which is used to separate a foreground object and a background object. A part of the image is divided into small blocks and explanation will now be made. Reference numeral 1001 denotes a block of the foreground object, 1002 a block of a boundary portion, and 1003 a block of a background portion. FIGS. 1B to 1D enlargedly show the blocks 1001 to 1003.
As will be understood from FIGS. 1B to 1D, values which are different from a value (data in the block 1001) which the foreground object has and a value (data in the block 1003) which the background object has exist in the boundary block 1002. FIG. 2 shows the luminance level of the image of this block on a line A-Axe2x80x2. In this example, the level from the luminance level of the foreground to the luminance level of the background smoothly changes.
According to the object extraction by the blue back, the value of the block 1003 corresponds to blue and the data at this level is removed as a background portion.
FIG. 3A shows a synthesized image obtained by superimposing another background into the background portion removed as mentioned above. FIGS. 3B to 3D are enlarged diagrams of the blocks 1001 to 1003. As will be understood from the block 1002 in FIG. 3C, even if the background object is replaced, a boundary area is in a state where data of the previous object is partially included. Therefore, discontinuous points are generated. FIG. 4 shows such a situation by the luminance level. In such a synthesized image, unnaturalness is conspicuous in an edge portion. Although a deviation of the luminance level causes a feeling of wrongness in the brightness of the edge, in case of a deviation of a color difference level, the edge is colored and the unnaturalness further increases.
To avoid such unnaturalness, a method whereby only a complete foreground object portion is extracted and synthesized to another background object without extracting data of a boundary area is also considered. FIG. 5A shows an example of such a case. FIGS. 5B to 5D enlargedly show the blocks 1001 to 1003. Since the data in the boundary area is not used, the foreground and background are clearly separated in the block 1002. FIG. 6 shows such a situation by the luminance level. According to an image obtained by merely simply superimposing the two objects as mentioned above, it feels as if an outline portion is visually emphasized. In this case as well, unnaturalness of the synthesized image is conspicuous.
A method of filtering the edge is also considered as an improved method of the above example. FIGS. 7A and 7B show examples in which a filtering process is performed to the image of FIG. 6. According to those examples, although the unnaturalness of the outline portion is reduced, since a width of boundary area to decide a degree of blur is unknown, FIG. 7A is the example in which the degree of blur is too smaller as compared with the original image and FIG. 7B shows the example in which the degree of blur is excessive.
According to the conventional method as mentioned above, it is extremely difficult to perform a natural image synthesization while accurately reproducing the boundary portion.
In consideration of the above circumstances, it is an object of the invention to provide an image processing apparatus and its method which can extract an object so as to obtain an accurate and natural image synthesization.
According to one preferred embodiment of the invention, there are provided an image processing apparatus and its method, wherein image data is inputted, the inputted image data is divided into blocks each constructed by a plurality of pixels, a motion of the image data is detected every block, and at least the image data of a first object and the image data of a second object are classified from the image data in accordance with a detection result.
According to another preferred embodiment of the invention, there are provided an image processing apparatus and its method, wherein image data is inputted, the image data is classified into at least a pixel of an area of a first object, a pixel of an area of a second object, and a pixel of a boundary area existing at a boundary between the area of the first object and the area of the second object, shape information to identify the area of the first object, the area of the second object, and the boundary area is formed, the classified image data and the formed shape information are encoded, and
wherein the shape information is information showing at which mixture ratio the pixels of the classified boundary area are constructed with the pixels of the area of the first object and the pixels of the area of the second object.
Other objects, features and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings.